Industrial internet of things, control methods and storage medium based on machine visual detection

ABSTRACT

The present disclosure discloses an industrial Internet of Things (IoT), a control method, and a storage medium based on a machine vision detection, and provides a technical solution for a timely adjustment of production line parameters according to image information generated during a machine vision data collection on the production line. Through a difference of the image information corresponding to different process operations, a processing situation of different process operations may be obtained, so that more accurate adjustment of the production line parameter may be achieved without increasing the system complexity, thereby effectively reducing a development cost of the industrial IoT, and increasing accuracy of the intelligent manufacturing control.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority of Chinese Patent Application No.202210983679.5, filed on Aug. 17, 2022, the contents of which areentirely incorporated herein by reference.

TECHNICAL FIELD

The present disclosure generally involves the field of industrialInternet of Things technology, and specifically involves an industrialInternet of Things (IoT), control methods and storage medium based onmachine vision detection.

BACKGROUND

A machine vision may be a technique and approach in industry to provideimage-based automated inspection and analysis for applications such asautomated inspection, process control, and robotic guidance, etc. Themachine vision technology may be a very important link on an industrialIoT. At present, the machine vision may be mainly applied in scenariosthat require material adjustment, such as a material grabbing,placement, and alignment. For the industrial IoT, an intelligent controlmainly depends on feedback of a parameter on a production line. Theintelligent control does not make full use of the data generated by themachine vision. Instead, too many sensors may increase a systemcomplexity.

SUMMARY

To at least overcome the above deficiencies in the existing technology,the purpose of the present disclosure is to provide an industrial IoT,control methods and storage medium based on a machine visual detection.

In the first aspect, the embodiment of the present disclosure providesthe industrial IoT based on machine visual detection, including: aservice platform, a management platform, and a sensing network platformthat interact in turn. The sensing network platform includes a sensingnetwork general platform and a plurality of sensing networksub-platforms.

The sensing network sub-platforms are configured to receive imageinformation of a product when a process operation of a production lineends as a first image data; each of the sensing network sub-platformsreceiving the first image data corresponding to different processoperation; identify first feature information of the first image data,and correct, according to the first feature information and baselinefeature information, the first image data to form second image data; andsend the second image data to the sensing network general platform.

The sensing network general platform is configured to receive the secondimage data, and sort the second image data according to the processoperation of the production line to form a first image database; andsend the first image database to the management platform.

The management platform is configured to identify a difference betweenadjacent second image data in the first image database as a first imagedifference data; generate, according to the first image difference data,a control parameter of a production line device, correct the controlparameter as a first parameter, issue the first parameter to theproduction line device through the sensing network platform; and sendthe first image database and the first image difference data to the userplatform for display through the service platform.

In existing technologies, the feed control of the intelligent adjustmentof the production line is mainly based on the data detected by therelevant sensor on the production line. However, in production practice,in order to obtain more accurate production line feedback, more sensorsare needed. However, increasing the number of sensors may not onlyincrease the complexity of the system, but also sometimes cause thesubsequent data processing overfitting, which reduces the controlaccuracy.

When the embodiments of the present disclosure are implemented, atechnical solution for a timely adjustment of production line parametersaccording to image information generated during a machine vision datacollection on the production line may be provided. Through thedifference of the image information corresponding to different processoperations, a processing situation of different process operations maybe obtained, so that more accurate adjustment of production lineparameter may be achieved without increasing the system complexity,thereby effectively reducing a development cost of the industrial IoT,and increasing accuracy of intelligent manufacturing control.

In the embodiment of the present disclosure, the sensing network generalplatform and the sensing network sub-platforms may be both configured asgateway servers, the service platform may be configured as a firstserver, and the management platform may be configured as a secondserver. The first image data received by the sensing networksub-platforms may usually be the image data used by the machine visionobtained when transferring, alignment, or other operations are performedat the end of a process operation. To improve efficiency of anidentification and correction, each sensing network sub-platformreceives and processes the first image data corresponding to differentprocess operations. The processing process may include identifying firstfeature information of the first image data, and performing an imagecorrection through baseline feature information preset in the sensingnetwork sub-platform. The first feature information identified by eachsensing network sub-platform may be the same or may be different, theembodiments of the present disclosure don't limit this. However, it isnecessary that the second image data formed after correction by the samesensor network sub-platform has the same product size and orientation.

In the embodiment of the present disclosure, the sensing network generalplatform may sort the process operations of the production line whenreceiving the second image data. A sorting approach may be performedaccording to ID of a shooting device of the second image data, or may beperformed according to the communication ID of the sensing networksub-platform., and the embodiment of the present disclosure doesn'tlimit this. An amount of calculation for sorting may be small, and anordinary intelligent gateway device may be fully capable of doing that.

In the embodiment of the present disclosure, the management platform maybe the most important platform for parameter processing, the managementplatform needs to identify a difference between the adjacent secondimage data in the first image database. Since the sensing networkgeneral platform generates the first image database according to a sortof the process operations of the production line, the managementplatform may obtain a change produced by any product at any processingstep after identifying the difference, and the change indicates theproduction process of the processing step. Through the detection of theproduction process, the control parameter of the production line may becorrected. At the same time, these images may further be displayed tothe user platform through the service platform.

In the second aspect, the embodiment of the present disclosure providesa control method for an industrial IoT based on a machine visiondetection. The method may be applied to the service platform, themanagement platform, and the sensing network platform that interacts inturn. The sensing network platform includes a sensing network generalplatform and a plurality of sensing network sub-platforms.

The method comprising: the sensing network sub-platforms receiving imageinformation of a product when a process operation of a production lineends as a first image data; each of the sensing network sub-platformsreceiving the first image data corresponding to different processoperation; the sensing network sub-platforms identifying first featureinformation of the first image data, and correcting, according to thefirst feature information and baseline feature information, the firstimage data to form second image data; the sensing network sub-platformssending the second image data to the sensing network general platform;the sensing network general platform receiving the second image data,and sorting the second image data according to the process operation ofthe production line to form a first image database; the sensing networkgeneral platform sending the first image database to the managementplatform; the management platform identifying a difference betweenadjacent second image data in the first image database as a first imagedifference data; the management platform generating, according to thefirst image difference data, the control parameter of a production linedevice, and correcting the control parameter as a first parameter, andissuing the first parameter to the production line device through thesensing network platform; and the management platform sending the firstimage database and the first image difference data to the user platformfor display through the service platform.

One of the embodiments of the present disclosure provides anon-transitory computer-readable storage medium storing computerinstructions, when reading the computer instructions in the storagemedium, a computer implements the control method for an industrial IoTbased on a machine vision detection.

Compared with the existing technology, the present disclosure has thefollowing advantages and beneficial effects.

The present disclosure provides a technical solution for a timelyadjustment of production line parameters according to image informationgenerated during a machine vision data collection on the production linebased on the industrial Internet of Things and control methods based onmachine visual detection. Through the difference of the imageinformation corresponding to different process operations, a processingsituation of different process operations may be obtained, so that moreaccurate adjustment of production line parameter may be achieved withoutincreasing the system complexity, thereby effectively reducing adevelopment cost of the industrial IoT, and increasing accuracy ofintelligent manufacturing control.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary embodiments of the present disclosure are described indetail with reference to the drawings. These embodiments arenon-limiting exemplary embodiments. In the drawings:

FIG. 1 is a schematic diagram illustrating structure of a systemaccording to some embodiments of the present disclosure;

FIG. 2 is a schematic diagram illustrating operations of a methodaccording to some embodiments of the present disclosure;

FIG. 3 is an exemplary flowchart illustrating a determining a secondimage difference data based on a first image difference data accordingto some embodiments of the present disclosure;

FIG. 4 is a schematic diagram illustrating determining a first parameterbased on an inversion model according to some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

To illustrate technical solutions of the embodiments of the presentdisclosure, a brief introduction regarding the drawings used to describethe embodiments is provided below. Obviously, the drawings describedbelow are merely some examples or embodiments of the present disclosure.Those having ordinary skills in the art, without further creativeefforts, may apply the present disclosure to other similar scenariosaccording to these drawings. It should be understood that the exemplaryembodiments are provided merely for better comprehension and applicationof the present disclosure by those skilled in the art, and not intendedto limit the scope of the present disclosure. Unless obvious accordingto the context or illustrated specifically, the same numeral in thedrawings refers to the same structure or operation.

In addition, the embodiments described are only a part of theembodiments of the present disclosure instead of all embodiments. Thecomponents of the embodiments described and shown in the drawings hereare usually arranged and designed by various configurations. Therefore,the detailed description of the embodiments of the present disclosureprovided in the drawings below does not mean to limit the scope of thepresent disclosure that requires protection, but only indicate theselected embodiments of the present disclosure. Based on the embodimentsof the present disclosure, all other embodiments obtained by thoseskilled in the art under the premise of not making creative labor belongto the scope of the present disclosure that requires protection.

To facilitate the explanation of the above industrial IoT based on amachine vision detection, please refer to FIG. 1 , FIG. 1 provides aschematic diagram of a communication structure of the industrial IoTbased on a machine vision detection disclosed by the embodiments of thepresent disclosure. The industrial IoT based on a machine visiondetection may include: a service platform, a management platform, and asensing network platform that interact in turn. The service platformfurther interacts with a user platform, which is used as a userterminal. The sensing network platform further interacts with an objectplatform, which is used as a production line, the sensing networkplatform includes a sensing network general platform and a plurality ofsensing network sub-platforms.

The sensing network sub-platforms may be configured to receive imageinformation of the product when a process operation of the productionline ends and take the image information as first image data; eachsensing network sub-platform receiving the first image datacorresponding to a different process operation; identify first featureinformation of the first image data, and correct the first image data toform second image data according to the first feature information andbaseline feature information; and send the second image data to thesensing network general platform.

The sensing network general platform may be configured to receive thesecond image data, and sort the second image data in accordance with theproduction line process operations to form a first image database; andsend the first image database to the management platform.

The management platform may be configured to identify a differencebetween the adjacent second image data in a first image database andtake the different as first image difference data; generate, accordingto the first image difference data, a control parameter of a productionline device, correct the control parameter of the production linedevice, and take the corrected control paremeter as a first parameter,and issue a first parameter through the sensing network platform to theproduction line device; and send the first image database and the firstimage difference data to the user platform for display through theservice platform.

In an existing technology, a feedback control of the intelligentadjustment of the production line may be mainly performed based on thedata detected by the relevant sensor on the production line. However, ina production practice, more sensors are needed to obtain more accurateproduction line feedback. However, increasing the number of sensors willnot only increase the complexity of the system, but also sometimescauses overfitting in subsequent data processing, which reduces controlaccuracy.

When the embodiments of the present disclosure are implemented, atechnical solution for a timely adjustment of production line parametersaccording to image information generated during a machine vision datacollection on the production line may be provided. Through thedifference of the image information corresponding to different processoperations, a processing situation of different process operations maybe obtained, so that more accurate adjustment of the production lineparameter may be achieved without increasing the system complexity,thereby effectively reducing a development cost of the Industrial IoT,and increasing accuracy of intelligent manufacturing control.

In the embodiment of the present disclosure, the sensing network generalplatform and the sensing network sub-platforms may be both configured asgateway servers, the service platform may be configured as a firstserver, and the management platform may be configured as a secondserver. The first image data received by the sensing networksub-platform may usually be the image data used by the machine visionobtained when transferring, alignment, or other operations are performedat the end of a process operation. To improve efficiency of anidentification and correction, each sensing network sub-platformreceives and processes the first image data corresponding to differentprocess operations. The processing process may include identifying firstfeature information of the first image data, and performing an imagecorrection through baseline feature information preset in the sensingnetwork sub-platform. The first feature information identified by eachsensing network sub-platform may be the same or may be different, theembodiments of the present disclosure don't limit this. However, it isnecessary that the second image data formed after correction by the samesensor network sub-platform has the same product size and orientation.

In the embodiment of the present disclosure, the sensing network generalplatform may sort the process operations of the production line whenreceiving the second image data. A sorting approach may be performedaccording to ID of a shooting device of the second image data, or may beperformed according to the communication ID of the sensing networksub-platform, and the embodiment of the present disclosure doesn't limitthis. An amount of calculation for sorting may be small, and an ordinaryintelligent gateway device may be fully capable of doing that.

In the embodiment of the present disclosure, the management platform maybe the most important platform for parameter processing, the managementplatform needs to identify a difference between the adjacent secondimage data in the first image database. Since the sensing networkgeneral platform generates the first image database according to a sortof the process operations of the production line, the managementplatform may obtain a change produced by any product at any processingstep after identifying the difference, and the change indicates theproduction process of the processing step. Through the detection of theproduction process, the control parameter of the production line may becorrected. At the same time, these images may further be displayed tothe user platform through the service platform.

In a possible implementation, the service platform includes a servicegeneral platform and a plurality of service sub-platforms.

The service sub-platform may be configured to receive the first imagedifference data; each service sub-platform receiving the first imagedifference data corresponding to different products of a same productionline; and transmit all the first image difference data to the servicegeneral-platform.

The service general platform is configured to calculate a difference ofcorresponding data in the first image difference data of differentproducts as a second image difference data; and send the second imagedifference data to the management platform.

The management platform is configured to correct, according to thesecond image difference data, the control parameter to form the firstparameter.

When the embodiment of the present disclosure is implemented,controlling the production parameter of the production line is not onlyto consider a processing situation of the product, but also consider astability of processed product. Therefore, in the embodiments of thepresent disclosure, the first image difference data of differentproducts on the same production line may be received through the servicesub-platform, and a comparison may be performed between the first imagedifference data through the service general platform. When the sameproduction line produces the same product, different products on thesame production line refer to different product individuals of the sameproduct. The difference of corresponding data in the first imagedifference data calculated by the service general platform refers to adifference data generated by different products in the same processoperation.

Exemplarily, there may be a continuous process operation A, processoperation B, and process operation C on the production line. At thistime, a product A is photographed through the process operation A toobtain an image AA, then the product A is photographed through theprocess operation B to obtain an image AB, and then the product A isphotographed through the process operation C to obtain an image AC. Aproduct B is photographed through the process operation A to obtain animage BA, then the product B is photographed through the processoperation B to obtain an image BB, and then the product B isphotographed through the process operation C to obtain an image BC. Thenthe first image difference data corresponding to the product A may be animage A1 and an image A2. The image A1 may be a non-calculating resultperformed by the image AB and the image AA. The image A2 may be anon-calculating result performed by the image AC and the image AB. Thefirst image difference data corresponding to the product B may be animage B1 and an image B2. The image B1 may be a non-calculating resultperformed by the image BB and the image BA. The image B2 may be anon-calculating result performed by the image BC and the image BB. Atthis time, the difference of the corresponding data in the first imagedifference data may be the difference between the image A1 and the imageB1, and the difference between the image A2 and the image B2.

The difference data generated in the same process operation of differentproducts indicates the stability of the process operation, that is, thestate of the product processed under the same production parameter.Therefore, in the embodiment of the present disclosure, the managementplatform may perform correction on the control parameter through thesecond image difference data, which may maintain the stability of theproduction line after issuing a corrected control parameter.

In some embodiments, the service general platform may further beconfigured to determine a time series based on the first imagedifference data. The time series may be a series of generation time ofeach product arranged in an order of time. A stationarity test may beperformed on the time series to determine whether the time series is astable series. In response to a determination that the time series is astable series, a first statistical data of the time series may becalculated, and determined as the second image difference data. Thefirst statistical data includes at least one of a sample moment and anautocovariance. In response to a determination that the time series isnot a stable series, a second statistical data of the time series may becalculated, and determined as the second image difference data. Thesecond statistical data includes at least one of the sample moment andthe autocovariance.

In some embodiments, the service general platform may further beconfigured to segment the time series and obtain at least one segmentedtime series, determine at least one second statistical data based on theat least one segmented time series, and integrate the at least onesecond statistical data to determine the second image difference data.The at least one second statistical data may be the at least one secondstatistical data corresponding to each segmented time series of the atleast one segmented time series.

In one possible implementation, the management platform may be furtherconfigured to compare the first image difference data and baseline imagedifference data configured on the management platform to generate athird image difference data; input the third image difference data to aninversion model to generate the control parameter, and input the secondimage difference data to the inversion model to generate a secondparameter; the input data of the inversion model may be the imagedifference data, and output data of the inversion model may be theproduction line parameter; and calculate a mean value of the controlparameter and the second parameter to form the first parameter.

When the embodiment of the present disclosure is implemented, theproduction line parameter may be calculated by a preset inversion model.The inversion model may be trained through a sample. It should beunderstood that the technology of model training through a sample in theexisting technology is very mature, and no matter what model is adopted,it should be within the protection range of the embodiments of thepresent disclosure. In the embodiment of the present disclosure, toreduce a calculation amount of the management platform, a baseline imagedifference data may be set up, so that the calculation may berespectively performed on the third image difference data and the secondimage difference data through only one inversion model, whicheffectively reduces model reading time of the management platform. Inthe embodiments of the present disclosure, the control parametercalculated through the third image difference data and the secondparameter calculated through the data of the second image differencedata need to be combined to generate the first parameter for issuing.Specifically, the first parameter may be generated by a mean valuecalculation method. In an operation process of the production line, thefirst parameter may be iterated and corrected. In the case of meeting amanufacturing technique requirement, the production process of theprocessing line may be stabilized as much as possible. It should beunderstood that the calculation of the mean value of the controlparameter and the second parameter mentioned in the embodiment of thepresent disclosure may be the mean value of the same parameter.

In some embodiments, the inversion model includes at least one of aninversion layer, a parameter fusion layer, and a difference predictionlayer. The inversion model may be a machine learning model. Themanagement platform may further be configured to process the third imagedifference data based on the inversion layer to generate the controlparameter, process the second image difference data based on theinversion layer to generate the second parameter, fuse the controlparameter and the second parameter based on the parameter fusion layerto determine the first parameter, and process the first parameter basedon the difference prediction layer to obtain a predicted third imagedifference data and/or a predicted second image difference data.

In some embodiments, a fusion weight of the control parameter and thesecond parameter in the parameter fusion layer may be related to anaverage proportion of preselected pixels in downstream image data.

In one possible implementation, the sensing network sub-platform mayfurther be configured to zoom and rotate the first feature informationto be fully aligned with the baseline feature information, record datacorresponding to the zooming and rotation as processing data; andcorrect the first image data according to the processing data to formthe second image data.

In one possible implementation, the management platform may further beconfigured to take image data at an upstream of the production line inthe adjacent second image data as upstream image data, and image data ata downstream of the production line in the adjacent second image data asdownstream image data; obtain edge data in the upstream image data asupper edge data, and obtain the edge data in the downstream image dataas lower edge data; filter out an edge that does not exist in the upperedge data from the lower edge data as a difference edge; select a pixelwhose a pixel value difference with pixel value of the upstream imagedata is greater than a preset value from the downstream image data as apreselected pixel, and filter out a pixel related to the difference edgefrom the preselected pixel to form the first image difference data. Thepixel related to the difference edge refers to the pixel located withina closed difference edge or between the difference edge and a graphicboundary.

When the embodiment of the present disclosure is implemented, a morespecific approach for obtaining difference data may be provided. In theproduction practice, the image applied by the machine vision lacks depthof field information, and lighting information of the image at differentlocations may be further different. To reduce an impact of the aboveinformation on the calculation of the difference data, an approach of acomprehensive identification of edge and difference pixel may beadopted. As an edge recognition is not affected by a difference oflighting, the identification of the difference edge may be generallymore accurate, and combined with portions of image identified by thepixel value, the specific first image difference data may be determined,which effectively improves the accuracy of the identification.

On the basis of the above, please refer to FIG. 2 , FIG. 2 is aflowchart illustrating control method for the industrial IoT based onmachine visual detection according to some embodiments of the presentdisclosure. The method may be applied in the industrial IoT based onmachine visual detection shown in FIG. 1 . Further, the method mayinclude the contents described in the following operations S1-S8.

S1: the sensing network sub-platforms receive image information of aproduct when a process operation of a production line ends as a firstimage data; each of the sensing network sub-platforms receiving thefirst image data corresponding to different process operation.

S2: the sensing network sub-platforms identify first feature informationof the first image data, and correct, according to the first featureinformation and baseline feature information, the first image data toform second image data.

S3: the sensing network sub-platforms send the second image data to thesensing network general platform.

S4: the sensing network general platform receives the second image data,and sort the second image data according to the process operation of theproduction line to form a first image database.

S5: the sensing network general platform sends the first image databaseto the management platform.

S6: the management platform identifies a difference between adjacentsecond image data in the first image database as a first imagedifference data.

S7: the management platform generates a control parameter of aproduction line device according to the first image difference data, andcorrects the control parameter as a first parameter, and issues thefirst parameter to the production line device through the sensingnetwork platform.

S8: the management platform sends the first image database and the firstimage difference data to a user platform for display through the serviceplatform.

In a possible implementation, the service platform includes a servicegeneral platform and a plurality of service sub-platforms.

The method further includes the service sub-platforms receiving thefirst image difference data; each service sub-platform receiving thefirst image difference data corresponding to different products of asame production line; the service sub-platforms transmitting all thefirst image difference data to the service general-platform; the servicegeneral-platform calculating a difference of corresponding data in thefirst image difference data of different products as a second imagedifference data; the service general-platform sending the second imagedifference data to the management platform; and the management platformcorrecting the control parameter to form the first parameter accordingto the second image difference data.

In a possible implementation, the management platform compares the firstimage difference data and baseline image difference data configured onthe management platform to generate a third image difference data.

The management platform inputs the third image difference data to aninversion model to generate the control parameter, and input the secondimage difference data to the inversion model to generate a secondparameter; the input data of the inversion model may be image differencedata, and output data of the inversion model may be production lineparameter.

The management platform calculates a mean value of the control parameterand the second parameter to form the first parameter.

In a possible implementation, the sensing network sub-platforms zoom androtate the first feature information to be fully aligned with thebaseline feature information, and record data corresponding to thezooming and rotation as processing data.

The sensing network sub-platforms correct the first image data to formthe second image data according to the processing data.

In a possible implementation, the management platform takes the imagedata at an upstream of the production line in the adjacent second imagedata as upstream image data, and image data at a downstream of theproduction line in the adjacent second image data as downstream imagedata.

The management platform obtains edge data in the upstream image data asupper edge data, and obtain the edge data in the downstream image dataas lower edge data; filter out an edge that does not exist in the upperedge data from the lower edge data as a difference edge.

The management platform selects a pixel whose a pixel value differencewith pixel value of the upstream image data is greater than a presetvalue from the downstream image data as a preselected pixel, and filterout a pixel related to the difference edge from the preselected pixel toform the first image difference data; the pixel related to thedifference edge refers to the pixel located within a closed differenceedge or between the difference edge and a graphic boundary.

FIG. 3 is an exemplary flowchart illustrating the determining secondimage difference data based on the first image difference data accordingto some embodiments of the present disclosure. In some embodiments, aflow 300 may be performed by a service general platform. As shown inFIG. 3 , the flow 300 includes the following operations.

In 310, a service general platform determining a time series based onthe first image difference data.

The time series may refer to a series of generation time of each productarranged according to the order of time. For example, the time seriesmay be {xt}, {yt}, etc. The generation time may refer to a time point ofeach product at the end of the corresponding process operation. Whenthere is a plurality of process operations, the time point at the end ofeach process operation corresponds to a generation time. For example,the first image difference data 1 is the difference between the image 1d after the correction obtained by photographing the product 1 at thetime point 1 through the process operation d and of the image 1e afterthe correction obtained by photographing the product 1 at the time point2 through the process operation e. The generation time of the processoperation e corresponding to the product 1 may be the time point 2. Thetime point in the time series corresponding to the first imagedifference data 1 may be the time point 2. As there is no processoperation before the first process operation in the production line,there is no corresponding first image difference data at the time pointof the end of the first process operation. The generation time does notinclude the time point at the end of the first process operation.

In some embodiments, the time series may reflect a plurality of firstimage difference data corresponding to a plurality of different timepoints arranged in the order of time, for example, the first imagedifference data of the same product through different process operationsat different time points, for another example, the first imagedifference data of different products through the same process operationat different time points.

In some embodiments, the first image difference data sent by the servicesub-platforms to the service general platform may include the generationtime of each product. The service general platform may arrange theplurality of first image difference data in the order of time accordingto the generation time of each product, and then determine the timeseries corresponding to the plurality of first image difference data.The service general platform may set a length of a time period of thetime series according to actual needs, for example, the time period maybe 30 minutes, 1 hour, 24 hours, 72 hours, etc. The service generalplatform may update the time period of the time series continuously. Forexample, the time series may be the time series corresponding to theplurality of first image difference data within 24 hours from thecurrent time point.

In 320, the service platform performing a stationarity test on the timeseries to determine whether the time series is a stable series.

The stable series may refer to a series where the first image differencedata in the time series basically has little change trend with time. Forexample, the changes of the plurality of first image difference data atdifferent time points basically fluctuate up and down within a certainrange. A mean value, variance, and covariance of the plurality of firstimage difference data corresponding to the stable series does not changewith time.

In some embodiments, the service general platform may perform astationarity test on the time series through a plurality of approaches(such as a graphic analysis, an assumption testing approach, etc.). Forexample, the service general platform may draw a line chart of arelationship between the first image difference data and the timeaccording to the time series. The service general platform may determinewhether the time series is stable through the line chart. The line chartmay reflect the changes of the plurality of first image difference dataat different time points. For example, when the line in the line chartfluctuates up and down at a certain value with the same fluctuationamplitude, the service general platform may determine that the timeseries is a stable series. For example, when the line in the line chartrises or decreases with time change, etc., the service general platformmay determine that the time series is a non-stable series. Thenon-stable series may refer to a series where the first image differencedata has features of trend, periodicity, and contingency, etc with timechange.

For another example, the service general platform may perform thestationarity test on the time series through a Dickey-Fuller Test and anAugmented Dickey-Fuller Test (ADF test). When the assumed value(P-Value) determined by the ADF test is less than a preset confidence,the service general platform may determine that the time series is astable series. When the assumed value (P-Value) determined by the ADFtest is greater than the preset confidence, the service general platformmay determine that the time series is not a stable series, i.e., anon-stable series. The preset confidence may be a preset maximumconfidence of the assumed value. For example, the preset confidence maybe 10%, 5%, 1%, etc.

In 330, in response to a determination that the time series is a stableseries, the service general platform calculating a first statisticaldata of the time series, and determining the first statistical data asthe second image difference data.

The first statistical data may refer to a statistical amount of theplurality of first image difference data corresponding to the stabletime series. In some embodiments, the first statistical data may berepresented in various ways. The first statistical data may include atleast one of a sample moment and an autocovariance. The sample momentmay include a k-order sample origin moment, a k-order sample centralmoment, etc. The autocovariance may include a k-order autocovariance,etc.

In some embodiments, the service general platform may calculate thefirst statistical data of the time series in various ways. For example,the service general platform may calculate the k-order sample originmoment of the time series through the formula (1):

$\begin{matrix}{{\overset{¯}{\alpha}}_{k} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}X_{i}^{k}}}} & (1)\end{matrix}$where α_(k) denotes the k-order sample origin moment of the time series;n denotes a number of samples of the time series; and X_(i) ^(k) denotesthe k-order sample origin moment of the ith sample.

For another example, the service general platform may calculate thek-order sample central moment of the time series through the formula(2):

$\begin{matrix}{{\overset{¯}{\beta}}_{k} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}( {X_{i} - \overset{\_}{X}} )^{k}}}} & (2)\end{matrix}$where β _(k) denotes the k-order autocovariance of the time series; ndenotes the number of samples of the time series; X_(i) denotes thefirst image difference data of the ith sample; and X denotes a meanvalue of the first image difference data of the n samples.

For another example, the service general platform may calculate thek-order autocovariance through formula (3) and formulas (4):γ_(k)≡Cov(y _(t) , y _(t+k))=E[(y _(t)−μ)(y _(t+k)−μ)]  (3)μ≡E(y)  (4)where γ_(k) denotes the k-order autocovariance of the time series y_(t),which reflects a degree of autocorrelation between a same variable y_(t)and a variable of interval k period; μ≡E(y) indicates a general meanvalue, and E indicates an expected value.

In some embodiments, the service general platform may determine thefirst statistical data (such as the k-order sample origin moment, thek-order sample central moment, and the k-order autocovariance, etc.)obtained through the above ways as the second image difference data.

In 340, in response to a determination that the time series is not astable series, the service general platform calculating a secondstatistical data of the time series, and determining the secondstatistical data as the second image difference data.

The second statistical data may refer to a statistical amount of theplurality of first image difference data corresponding to all or aportion of the non-stable series. In some embodiments, the secondstatistical data may be represented in various ways. The secondstatistical data includes at least one of a sample moment and anautocovariance.

In some embodiments, when the time series is a non-stable series, theservice general platform may further determine the time periodcorresponding to the non-stable series in the time series. The timeseries may be divided into a first time period corresponding to thestable series and a second time period corresponding to the non-stableseries. Through the approach of determining the first statistical datain 330, the service general platform may respectively determine secondstatistical data 1 corresponding to the first time period and secondstatistical data 2 corresponding to the second time period. The servicegeneral platform may determine the second statistical data 1 and thesecond statistical data 2 as the second image difference data.

In some embodiments, when the time series is a non-stable series, theservice general platform may segment the time series and obtain at leastone segmented time series. The service general platform may determine atleast one second statistical data based on the at least one segmentedtime series. The service general platform may integrate the at least onesecond statistical data to determine the second image difference data.

The segmented time series may include the time series corresponding toat least one time period.

In some embodiments, the service general platform may segment the timeseries in various ways. For example, the service general platform maysegment the time series randomly. For another example, the servicegeneral platform may segment the time series according to a preset rule.The preset rule may specify a number of the segment, a principle of thesegment (such as average segment, etc.), the minimum time length of asegment, etc.

In some embodiments, after the first segment, at least one segmentedtime series may include the stable series and the non-stable series. Forthe at least one segmented time series, the service platform may repeatoperation 320 to determine whether the at least one segmented timeseries is a stable series. When a certain segmented time series is thestable series, the segmented time series may not be further segmented.When a certain segmented time series is the non-stable series, theservice general platform may continue to perform a second segment. Theservice general platform may continue to repeat operation 320 tore-determine whether the at least one second segmented time series isthe stable series. The service general platform stops segmenting untilthe at least one segmented time series is a stable series. The servicegeneral platform obtains at least one final segmented time seriescorresponding to the time series.

In some embodiments, when the segmented time series is less than a timeseries threshold, the service general platform stops further segmentingthe segmented time series. The time series threshold may refer to theminimum value of the time series. The service general platform maypreset the time series threshold in advance. When the segmented timeseries less than the time series threshold is a non-stable series, theservice general platform may label the segmented time series todistinguish the segmented time series from other segmented time seriesbeing stable series so as to facilitate the subsequent separatestatistics of the segmented time series of stable series and thesegmented time series of non-stable series.

Each segmented time series may correspond to a second statistical data.The service general platform may respectively determine at least onesecond statistical data corresponding to at least one segmented timeseries based on at least one segmented time series. The determining atleast one second statistical data based on the at least one segmentedtime series may be referred to the operation 330, for example, theservice general platform calculates the first statistical data of thetime series in operation 330, the service general platform may calculatethe second statistical data corresponding to each segmented time seriesaccording to operation 330. Through the calculation, the service generalplatform may determine a plurality of second statistical datacorresponding to the plurality of segmented time series.

In some embodiments, the service general platform may integrate (such asweighting or averaging) the plurality of second statistical datarespectively corresponding to the plurality of segmented time series todetermine the second image difference data. When the plurality ofsegmented time series include non-stable series, the service generalplatform may integrate the plurality of second statistical datacorresponding to the stable series to determine the second imagedifference data corresponding to the stable series. The service generalplatform may stitch the second statistical data corresponding to thestable series and the second statistical data corresponding to thenon-stable series to determine final second image difference data.

In some embodiments of the present disclosure, when the time series is anon-stable series, the service general platform may segment the timeseries. The service general platform determining the plurality of secondstatistical data respectively corresponding to the plurality ofsegmented stable series may further improve the accuracy of the finaldetermined second image difference data, which helps to realize moreaccurately adjustment of subsequent production line parameter.

In some embodiments of the present disclosure, the service generalplatform may determine the calculated statistical data of the timeseries of the first image difference data as the second image differencedata. In this way, the accuracy of the final determined second imagedifference data may be improved, which helps to realize more accuratelyadjustment of subsequent production line parameter.

FIG. 4 is a schematic diagram illustrating a determining a firstparameter based on an inversion model according to some embodiments ofthe present disclosure. In some embodiments, a flow 400 may be performedby the management platform.

In some embodiments, the inversion model 430 may include at least one ofan inversion layer 431, a parameter fusion layer 432, and a differenceprediction layer 433. The inversion model may be a machine learningmodel.

The inversion layer may be used to process image difference data todetermine a production line parameter. The inversion layer 431 mayinclude a first inversion layer 4311, and a second inversion layer 4312,etc. In some embodiments, different inversion layers may processdifferent image difference data to generate different production lineparameters. For example, the management platform may process third imagedifference data 410 based on the first inversion layer 4311 to generatea control parameter 440. Exemplarily, an input of the first inversionlayer 4311 may be the third image difference data 410, and an output ofthe first inversion layer 4311 may be the control parameter 440. Foranother example, the management platform may process second imagedifference data 420 based on the second inversion layer 4312 to generatea second parameter 450. Exemplarily, the input of the second inversionlayer 4312 may be the second image difference data 420, and the outputof the second inversion layer 4312 may be the second parameter 450. Formore descriptions of the third image difference data, the controlparameter, and the second parameter, please refer to relateddescriptions of FIG. 1 and FIG. 2 .

The parameter fusion layer 432 may be configured to fuse differentproduction line parameters to determine the first parameter. In someembodiments, the management platform may fuse the control parameter andthe second parameter based on the parameter fusion layer to determinethe first parameter. For example, the input of the parameter fusionlayer 432 may be the control parameter 440 and the second parameter 450,and the output of the parameter fusion layer 432 may be the firstparameter 460. For more descriptions of the first parameter, pleaserefer to the related descriptions of FIG. 1 and FIG. 2 .

In some embodiments, the management platform may determine the firstparameter based on a fusion weight of the control parameter and thesecond parameter in the parameter fusion layer. The fusion weight of thecontrol parameter and the second parameter in the parameter fusion layermay be related to an average proportion of preselected pixels indownstream image data. For more descriptions of the preselected pixelsand downstream image data, please refer to related descriptions of FIG.1 and FIG. 2 .

The fusion weight may refer to weight coefficients respectivelycorresponding to the control parameter and the second parameter duringthe fusion.

A preselected pixel corresponds to a proportion of downstream imagedata. The proportion may refer to a ratio of a number of preselectedpixels to a total number of pixels in the downstream image data. Themanagement platform may obtain the corresponding proportions of aplurality of preselected pixels in a plurality of downstream image data.The management platform may determine the average proportion ofpreselected pixels in downstream image data by averaging.

In some embodiments, the fusion weight corresponding to the controlparameter may be proportional to the average proportion of thepreselected pixels in the downstream image data. The fusion weightcorresponding to the second parameter may be inversely proportional tothe average proportion of the preselected pixels in the downstream imagedata. The average proportion of the preselected pixels in the downstreamimage data is large, indicating that the first image data difference ofthe product is great, a processing intensity of the correspondingprocess operations on the production line may be relatively large, andthe product may change more significantly after the process operations.As a result, an acceptability of the difference between thecorresponding second image difference data may be high, and the fusionweight corresponding to the second parameter determined by the secondimage difference data may be reduced.

In some embodiments of the present disclosure, determining the fusionweight through the average proportion of preselected pixels in thedownstream image data may further improve the accuracy of the firstparameter determined by the parameter fusion layer.

The difference prediction layer 433 may be configured to predict theimage difference data based on a first parameter. In some embodiments,the management platform may process the first parameter based on thedifference prediction layer to obtain a predicted third image differencedata and/or a predicted second image difference data. The input of thedifference prediction layer 433 may include the first parameter 460, andthe output of the difference prediction layer 433 may include thepredicted third image difference data 470 and/or the predicted secondimage difference data 480. The predicted third image difference data mayrefer to difference data between the first image difference data andbaseline image difference data predicted by the difference predictionlayer based on the first parameter. The predicted second imagedifference data may refer to difference data in the first imagedifference data of different products predicted by the differenceprediction layer based on the first parameter.

In some embodiments, when determining the first parameter based on theinversion model, the third image difference data 410 and the secondimage difference data 420 may be input to the inversion model. In theinversion model, the management platform may input the third imagedifference data 410 into the first inversion layer 4311. The controlparameter 440 output by the first inversion layer 4311 may be used asthe input of the parameter fusion layer 432. In the inversion model, themanagement platform may input the second image difference data 420 tothe second inversion layer 4312. The second parameter 450 output by thesecond inversion layer 4312 may be used as the input of the parameterfusion layer 432. In the inversion model, the management platform mayinput the control parameter 440 and the second parameter 450 to theparameter fusion layer 432. The first parameter 460 output by theparameter fusion layer 432 may be used as the output of the inversionmodel 430. The inversion layer, the parameter fusion layer, and thedifference prediction layer may be a neural network or a structure ofother machine learning models, the implementation approaches andstructures of which may be different.

The inversion model may be obtained through training. The differenceprediction layer in the inversion model may be obtained by pre-trainingbased on a training sample, and may be optimized when the inversionlayer and the parameter fusion layer are jointly trained. During thepre-training of the difference prediction layer, the training sample mayinclude a historical first parameter. The training sample may beobtained based on historical data. A label may be historical third imagedifference data and historical second image difference datacorresponding to a historical first parameter.

The inversion layer and the parameter fusion layer in the inversionmodel may be obtained by jointly training with the pre-traineddifference prediction layer.

During the training, the input of the inversion layer may come from thethird image difference data and the second image difference data inactual historical data. The sample label may be an actual controlparameter and the actual second parameter in the historical data. Duringthe training, the input of the parameter fusion layer may come from theactual control parameter and the actual second parameter in thehistorical data. The sample label may be the actual first parameter inthe historical data. Thus, the management platform may train a model toobtain the first parameter corresponding to a target condition based ona law among the types of data. In this way, a complex calculation anduncertainty of a trial calculation through a plurality of sets of presetparameters may be reduced.

During the training, a loss function may include a loss item related tothe predicted first parameter, a loss item of the third image differencedata based on a first compliance rate of the difference data, and a lossitem of the second image difference data based on a second compliancerate of the difference data. The joint training may be performed invarious feasible approaches, such as gradient descent.

The first compliance rate of the difference data may refer to the degreeof compliance between the predicted third image difference data and thethird image difference data. For example, the first compliance rate ofthe difference data may be determined based on a difference between thepredicted third image difference data and the third image differencedata. Exemplarily, the first compliance rate of the difference data maybe inversely proportional to the difference between the predicted thirdimage difference data and the third image difference data. The smallerthe difference between the predicted third image difference data and thethird image difference data is, the greater the first compliance rate ofthe difference data is.

The second compliance rate of the difference data may refer to thedegree of compliance between the predicted second image difference dataand the second image difference data. For example, the second compliancerate of the difference data may be determined based on the differencebetween the predicted second image difference data and the second imagedifference data. Exemplarily, the second compliance rate of thedifference data may be inversely proportional to the difference betweenthe predicted second image difference data and the second imagedifference data. The smaller the difference between the predicted secondimage difference data and the second image difference data is, thegreater the second compliance rate of the difference data is.

Through the above model, the first parameter that meets a requirementmay be predicted based on experience learned from the historical data.The model may predict the image difference data that may be actuallyobtained based on the obtained first parameter. With the combination ofthe loss function, the model may make the actual image difference datacloser or better than the image difference data.

The inversion model provided by some embodiments of the presentdisclosure may determine the first parameter, and input the determinedfirst parameter to the difference prediction layer to obtain thepredicted third image difference data and the predicted second imagedifference. In this way, a more ideal first parameter may be obtained,which improves the accuracy of the inversion model, and helps toimproving accuracy of the intelligent manufacturing control.

Those skilled in the art may realize that the units and algorithmsdescribed in the embodiments of the present disclosure may beimplemented through an electronic hardware, a computer software, or thecombination thereof. To clearly explain the interchangeability of thehardware and the software, the above has described the composition andoperation of each embodiment in general. Whether these functions areimplemented by the hardware or the software depends on the specificapplication and design constraints of the technical solution. Thoseskilled in the art may use different approaches to implement thedescribed functions on each specific application, but thisimplementation should not be considered to exceed the scope of thepresent disclosure.

In several embodiments provided by the present disclosure, it should beunderstood that the device and method disclosed should be implemented inother ways. For example, the embodiment of the device described above isonly for the purpose of illustration. For example, the division of theunit is only one kind of logical function division. In practice, theremay be another way to divide. For example, a plurality of units orcomponents may be combined or integrated to another system, or somefeatures may be ignored or not executed. In addition, the shown ordiscussed mutual coupling or direct coupling or communication connectionmay be indirect coupling or communication connection through someinterfaces, devices, or units, or may be electrical, mechanical, orother forms of connection.

The units that are described as a separate part may be separatedphysically or not. As a unit, it is obvious for those skilled in the artthat the units and algorithms described in the embodiments of thepresent disclosure may be implemented through an electronic hardware, acomputer software, or the combination thereof. To clearly explain theinterchangeability of the hardware and the software, the above hasdescribed the composition and operation of each embodiment in general.Whether these functions are implemented by the hardware or the softwaredepends on the specific application and design constraints of thetechnical solution. Those skilled in the art may use differentapproaches to implement the described functions on each specificapplication, but this implementation should not be considered to exceedthe scope of the present disclosure.

In addition, each functional unit in each embodiment of the presentdisclosure may be integrated in one processing unit, or they mayphysically exist independently, or two or more units may be integratedin one unit. The above integrated units may be implemented in the formof hardware or the form of software functional units.

When the integrated unit is implemented in the form of a softwarefunctional unit or used or sold as an independent product, it may bestored in a computer readable storage medium. Based on thisunderstanding, the technical solution of the present disclosure isessentially or the part that contributes to the prior art, or all orpart of the technical solution can be embodied in the form of a softwareproduct, and the computer software product is stored in a storagemedium, including several instructions used to make a computer device(which may be a personal computer, a server, or a grid device, etc.) toexecute all or part of the operations of the methods described in thevarious embodiments of the present disclosure. The aforementionedstorage medium includes: a U disk, a removable hard disk, a Read-OnlyMemory (ROM, Read-Only Memory), a Random Access Memory (RAM, RandomAccess Memory), a magnetic disk or optical disk and other media that canstore program codes.

The specific embodiments described above further describe the purpose,technical solutions, and beneficial effects of the present disclosure indetail. It should be understood that the above descriptions are onlyspecific embodiments of the present disclosure, and are not intended tolimit the scope of the present disclosure. The protection scope, anymodification, equivalent replacement, improvement, etc. made within thespirit and principle of the present disclosure shall be included in theprotection scope of the present disclosure.

What is claimed is:
 1. An industrial Internet of Things (IoT) systembased on a machine vision detection, including: a service platform, amanagement platform, and a sensing network platform that interact inturn, wherein the sensing network platform includes a sensing networkgeneral platform and a plurality of sensing network sub-platforms;wherein: the sensing network sub-platforms are configured to: receiveimage information of a product when a process operation of a productionline ends as a first image data; each of the sensing networksub-platforms receiving the first image data corresponding to differentprocess operation; identify first feature information of the first imagedata, and correct, according to the first feature information andbaseline feature information, the first image data to form second imagedata; send the second image data to the sensing network generalplatform; the sensing network general platform is configured to: receivethe second image data, and sort the second image data according to theprocess operation of the production line to form a first image database;send the first image database to the management platform; the managementplatform is configured to: identify a difference between adjacent secondimage data in the first image database as a first image difference data;generate, according to the first image difference data, a controlparameter of a production line device, and correct the control parameteras a first parameter, and issue the first parameter to the productionline device through the sensing network platform; and send the firstimage database and the first image difference data to the user platformfor display through the service platform.
 2. The industrial IoT systemof claim 1, wherein the service platform includes a service generalplatform and a plurality of service sub-platforms, wherein: the servicesub-platforms are configured to: receive the first image differencedata; each service sub-platform receiving the first image differencedata corresponding to different products of a same production line;transmit all the first image difference data to the service generalplatform; the service general platform is configured to: calculate adifference of corresponding data in the first image difference data ofdifferent products as a second image difference data; and send thesecond image difference data to the management platform, the managementplatform correcting, according to the second image difference data, thecontrol parameter to form the first parameter.
 3. The industrial IoTsystem of claim 2, wherein the management platform is further configuredto: compare the first image difference data and baseline imagedifference data configured on the management platform to generate athird image difference data; input the third image difference data to aninversion model to generate the control parameter, and input the secondimage difference data to the inversion model to generate a secondparameter; wherein input data of the inversion model is image differencedata, and output data of the inversion model is production lineparameter; and calculate a mean value of the control parameter and thesecond parameter to form the first parameter.
 4. The industrial IoTsystem of claim 1, wherein the sensing network sub-platforms are furtherconfigured to: zoom and rotate the first feature information to be fullyaligned with the baseline feature information, and record datacorresponding to the zooming and rotation as processing data; andcorrect, according to the processing data, the first image data to formthe second image data.
 5. The industrial IoT system of claim 1, whereinthe management platform is further configured to: take image data at anupstream of the production line in the adjacent second image data asupstream image data, and image data at a downstream of the productionline in the adjacent second image data as downstream image data; obtainedge data in the upstream image data as upper edge data, and obtain theedge data in the downstream image data as lower edge data; filter out anedge that does not exist in the upper edge data from the lower edge dataas a difference edge; and select, from the downstream image data, apixel whose a pixel value difference with pixel value of the upstreamimage data is greater than a preset value as a preselected pixel, andfilter out, from the preselected pixel, a pixel related to thedifference edge to form the first image difference data; wherein thepixel related to the difference edge refers to the pixel located withina closed difference edge or between the difference edge and a graphicboundary.
 6. A control method for an industrial Internet of Things (IoT)system based on a machine vision detection, which is used in a serviceplatform, a management platform, and a sensing network platform thatinteract in turn, wherein the sensing network platform includes asensing network general platform and a plurality of sensing networksub-platforms; the method comprising: the sensing network sub-platformsreceiving image information of a product when a process operation of aproduction line ends as a first image data; each of the sensing networksub-platforms receiving the first image data corresponding to differentprocess operation; the sensing network sub-platforms identifying firstfeature information of the first image data, and correcting, accordingto the first feature information and baseline feature information, thefirst image data to form second image data; the sensing networksub-platforms sending the second image data to the sensing networkgeneral platform; the sensing network general platform receiving thesecond image data, and sorting the second image data according to theprocess operation of the production line to form a first image database;the sensing network general platform sending the first image database tothe management platform; the management platform identifying adifference between adjacent second image data in the first imagedatabase as a first image difference data; the management platformgenerating, according to the first image difference data, a controlparameter of a production line device, and correcting the controlparameter as a first parameter, and issuing the first parameter to theproduction line device through the sensing network platform; and themanagement platform sending the first image database and the first imagedifference data to the user platform for display through the serviceplatform.
 7. The method of claim 6, wherein the service platformincludes a service general platform and a plurality of servicesub-platforms; the method further comprising: the service sub-platformsreceiving the first image difference data; each service sub-platformreceiving the first image difference data corresponding to differentproducts of a same production line; the service sub-platformstransmitting all the first image difference data to the service generalplatform; the service general platform calculating a difference ofcorresponding data in the first image difference data of differentproducts as a second image difference data; the service general platformsending the second image difference data to the management platform; andthe management platform correcting, according to the second imagedifference data, the control parameter to form the first parameter. 8.The method of claim 7, wherein the management platform compares thefirst image difference data and baseline image difference dataconfigured on the management platform to generate a third imagedifference data; the management platform inputs the third imagedifference data to an inversion model to generate the control parameter,and input the second image difference data to the inversion model togenerate a second parameter; wherein input data of the inversion modelis image difference data, and output data of the inversion model isproduction line parameter; and the management platform calculates a meanvalue of the control parameter and the second parameter to form thefirst parameter.
 9. The method of claim 6, wherein the sensing networksub-platforms zoom and rotate the first feature information to be fullyaligned with the baseline feature information, and record datacorresponding to the zooming and rotation as processing data; and thesensing network sub-platforms correct, according to the processing data,the first image data to form the second image data.
 10. The method ofclaim 6, wherein the management platform takes image data at an upstreamof the production line in the adjacent second image data as upstreamimage data, and image data at a downstream of the production line in theadjacent second image data as downstream image data; the managementplatform obtains edge data in the upstream image data as upper edgedata, and obtains the edge data in the downstream image data as loweredge data; filters out an edge that does not exist in the upper edgedata from the lower edge data as a difference edge; and the managementplatform selects, from the downstream image data, a pixel whose a pixelvalue difference with pixel value of the upstream image data is greaterthan a preset value as a preselected pixel, and filters out, from thepreselected pixel, a pixel related to the difference edge to form thefirst image difference data; wherein the pixel related to the differenceedge refers to the pixel located within a closed difference edge orbetween the difference edge and a graphic boundary.